1. Field of the Invention
The present invention relates to a technique of adjusting the position/orientation of a virtual object combined with a physical object.
2. Description of the Related Art
Many systems have been proposed which provide a natural mixture of a physical space and a virtual space based on mixed reality technology. In these mixed reality systems, an image in a virtual space drawn using a computer graphics (CG) technique is superimposed on an image in a physical space taken using an image pickup apparatus such as a camera, and the resultant mixed image is displayed on a display such as a head-mounted display (MHD) thus presenting a mixed reality image to a user.
In recent years, great advances in three-dimensional CAD (Computer Aided Design) techniques and rapid prototyping techniques have been made, and it has become possible to produce, automatically and in a short time, a mock-up of a physical object based on shape model data produced by a three-dimensional CAD system on a computer.
However, although actual mock-up models produced using a rapid prototyping shaping apparatus (hereinafter such models will be referred to as “physical models”) can be similar in terms of shape to shape models produced using a three-dimensional CAD system (hereinafter, such models will be referred to as “virtual models”), materials of mock-up models are limited to those used in shaping. Therefore, all characteristics such as colors, textures, and patterns of virtual models cannot be reflected in physical models. To avoid the above problem, it is known to draw a virtual model, based on which to produce a physical model, using CG and a mixed reality system, and display the virtual model superimposed on the physical model thus allowing it to evaluate a design in terms of characteristics such as colors, textures, and patterns that cannot be evaluated when the conventional rapid prototyping technique is used.
In such a system, it is required to display a virtual model in a virtual space such that the virtual model exactly coincides with a physical model existing in a physical space. In this technique, because a physical model is produced from a virtual model produced using a three-dimensional CAD system, both models are identical in terms of the shape and the size. However, to achieve coincidence in terms of the position/orientation between both models in the mixed reality technique, it is required not only to achieve exact consistency between the physical space and the virtual space, but it is also required to place the virtual model in the virtual space and the physical model in the physical space such that the position/orientation of the physical model placed in the physical space exactly coincides with the position/orientation of the virtual model placed in the virtual space. That is, it is required to first set a coordinate system for the physical space and a coordinate system for the virtual space such that they exactly coincide with each other, and then to place the physical model and the virtual model such that their corresponding coordinates in the respective coordinate systems are exactly equal to each other.
To achieve the first requirement in terms of the consistency between the physical space and the virtual space, many positioning techniques have been proposed (an example may found, for example, in U.S. Published application No. 2002095265 or Japanese Patent Laid-Open No. 2003-269913).
To achieve the second requirement, it is known to measure the position/orientation of the physical model in advance, and place the virtual model in accordance with the measured position/orientation of the physical model.
One known method of measuring the position/orientation of the physical model is to use a position/orientation measuring device, such as a magnetic sensor to measure a three-dimensional position/orientation using magnetism. Another known method is to stick a plurality of markers on a physical model, at known three-dimensional positions, and take an image of the physical model using an image pickup apparatus such as a camera. Markers are extracted from the image, and the position/orientation of the physical model is calculated from the correspondence between the coordinates of the extracted makers in the image and the three-dimensional positions/orientations of the markers.
However, sensors are generally expensive. In a case where the cost for a system is limited, it is difficult to use a sufficiently large number of sensors. This is a problem to be solved. Depending on the size of a physical model, it is difficult to attach three-dimensional position/orientation sensors, such as a magnetic sensor, to the physical model.
When a design is evaluated, it is necessary to perform the evaluation for various environmental conditions such as lighting in a physical space. However, depending on the environmental conditions, it is difficult to correctly detect the color of markers stuck on a physical model. In a case where a system is placed for exhibition in a showroom, colors and/or shapes of markers stuck on a physical model are limited to those that do not spoil the beauties of the showroom, and thus it is not necessarily possible to use desirable colors and/or shapes.
When it is difficult to directly measure the position/orientation of a physical model, it is known to define the position/orientation of a physical model and set the physical model at the defined position/orientation. However, even in this technique, an error in the position/orientation of the physical model often occurs. To minimize the error, after the physical model is set at the defined position/orientation, the position/orientation of a virtual model is manually adjusted in a trial-and-error fashion until the position/orientation of the physical model becomes consistent with the position/orientation of the virtual model.
In a common mixed reality system, an image in a virtual space is displayed such that it is superimposed on an image in a physical space. Therefore, for a user of the system, a virtual model is viewed in front of a physical model, and it is difficult for the user to recognize the correct relative positions of the physical model and the virtual model. This problem is serious, in particular, when the fine adjustment is made on the position/orientation of the virtual model, because, in the fine adjustment, it is necessary to detect the exact position/orientation for both the physical model and the virtual model.